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Claims 

We Claim: 

1. A method of multiattribute analysis and optimization for providing automated 
measurements of the importance of attributes and attribute values of message 
alternatives and for, in an automated manner, preferentially in real-time, improving 
the stage-to-stage performance of said message alternatives in a multi-stage message 
campaign in an interactive measurable medium; said method including: 

obtaining historical information including historical performance data for 
message performance for at least one previous stage of said campaign; 

generating a multiattribute system that describes said message alternatives in 
terms of said attributes and said attribute values; 

determining the importance of said attributes and said attribute values to the 
performance of said message alternatives; 

optimizing campaign performance through allocation of message alternatives 
to visitors during said message campaign; and 

determining a next allocation for each new visitor for a next stage of the 
campaign. 

2. A method as in claim 1, further comprising: processing said historical performance 
data to identify erroneous or possibly erroneous or unusual historical performance 
data. 

3. A method as in claim 2, wherem said processing further including cleaning said 
historical performance data to correct said identified erroneous or possible erroneous 
or unusual historical performance data. 
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4. A method as in claim 3, further comprising discounting at least a portion of said 
historical performance data from said at least one previous stage to grant more weight 
to more recent historical performance data than to less recent historical performance 
data. 

5. A method as in claim 1, further comprising discounting at least a portion of said 
historical performance data from said at least one previous stage to grant more weight 
to more recent historical performance data than to less recent historical performance 
data. 

6. A method as in claim 5, wherein said historical information further includes historical 
information selected from the set consisting of a visitor identifier, a message 
alternative identifier for the message alternative shown to said visitor, a type of action 
or types of actions taken by each said visitor, and a payoff for each of the actions 
taken by said visitors. 

7. A method as in claim 1, wherein said historical information fiirther includes historical 
information selected from the set consisting of a visitor identifier, a message 
alternative identifier for the message alternative shown to said visitor, a type of action 
or actions taken by each said visitor, and a payoff for each of the action or actions 
taken by said visitors. 

8. A method as in claim 6, wherein said historical information further includes 
additional information selected from the set consisting of: a message type, a message 
type and a corresponding alternative clicked on by a visitor, a. web site and a section 
of the web site where the visitor was presented with and saw or had an opportunity to 
see a banner ad, a time of the visitors visit, a visitor demographic information, a 
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visitor psychographic information, a visitor demographic profile, a visitor 
psychographic profile, and combinations thereof. 

9. A method as in claim 7, wherein said historical information further includes 
additional information selected from the set consisting of: a message type, a message 
type and a corresponding alternative clicked on by a visitor, a web site and a section 
of the web site where the visitor was presented with and saw or had an opportunity to 
see a banner ad, a time of the visitors visit, a visitor demographic information, a 
visitor psychographic information, a visitor demographic profile, a visitor 
psychographic profile, and combinations thereof 

10. A method as in claim 1, wherein said historical information includes additional 
information selected from the set consisting of: a message type, a message type and a 
corresponding alternative clicked on by a visitor, a web site and a section of the web 
site where the visitor was presented with and saw or had an opportunity to see a 
banner ad, a time of the visitors visit, a visitor demographic information, a visitor 
psychographic information, a visitor demographic profile, a visitor psychographic 
profile, and combinations thereof. 

1 1 . A method as in claim 1, fijrther comprising: 

processing said historical data to identify and correct any erroneous data; 

discounting at least a portion of said performance data fi-om said at least one 
previous stage to grant more weight to more recent performance data than to 
less recent performance data; and 

said historical information further includes: information selected from the set 
consisting of: a visitor identifier, a message alternative identifier for the 
message alternative shown to said visitor, a type of action or actions taken by 
said visitor, and a payoff for each of the actions taken by said visitors, a 
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message type, a message type and a corresponding alternative clicked on by a 
visitor, a web site and a section where the visitor was presented and saw a 
banner ad, a time of the visitors visit, a visitor demographic information, a 
visitor psychographic information, a visitor demographic profile, a visitor 
5 psychographic profile, and combinations thereof 

12. A method as in claim 1 1, wherein said message alternative is selected from a group of 
message alternatives consisting of: an email, an ad, a banner ad, a banner, a splash 
, ^ page, a home page, a jump page, a landing page, media programming, media content, 

CI 10 a political campaign message, a survey, a poll, a news headline, a headline, a ballot 
I'll measure, a ballot initiative, one or more sports scores, one or more sports scores for a 

% local, regional, collegiate, or amateur sports team or teams, and combinations thereof 

'SS? 

r . 13. A method as in claim 1, wherein said message alternative is selected fi-om a group of 

111 15 message altematives consisting of: an email, an ad, a banner ad, a splash page, a 
%| home page, a jump page, a landing page, media programming, media content, a 

political campaign message, a survey, a poll, a news headhne, a headline, a ballot 
measure, a ballot initiative, one or more sports scores, one or more sports scores for a 
local, regional, collegiate, or amateur sports team or teams, and combinations thereof. 

20 

14. A method as in claim 1, wherein each exposure of a message alternative to a visitor 
resuhs in a payoff result, a performance of said message alternative for a visitor 
segment being based at least in part on the expected value of tiie payoff results for the 
visitors for that visitor segment, said payoff result being a function of a measurement 
25 including at least one measurement selected from the set of measurements consisting 
of: a number of actions, a number of signups, a number of purchases, a binary result, 
a value of purchases, a revenue amount, a sales amount, a profit amount, a continuous 
function of a continuous variable, a continuous function of a discontinuous variable, a 
continuous function of a binary variable, a discrete function of a continuous variable, 
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a discrete function of a discontinuous variable, a discrete function of a binary 
variable, and combinations tiiereof 



15. A method as in claim 12, wherein each exposure of a message alternative to a visitor 
results in a payoff result, a performance of said message alternative for a visitor 
segment being based at least in part on the expected value of the payoff results for the 
visitors for that visitor segment, said payoff result being a function of a measurement 
including at least one measurement selected from the set of measurements consisting 
of: a number of actions, a number of signups, a number of purchases, a binary result, 
a value of purchases, a revenue amount, a sales amount, a profit amount, a continuous 
function of a continuous variable, a continuous function of a discontinuous variable, a 
continuous function of a binary variable, a discrete function of a continuous variable, 
a discrete function of a discontinuous variable, a discrete function of a binary 
variable, and combinations thereof. 

16. A method as in claim 14, wherein said multiattribute method comprises: 

generating at least one report on the optimal value for each attribute; 

generating new message alternatives with the values of each attribute indicated by 
a multiattribute analysis that uses prior expectations and payoff criteria to arrive at 
message alternative evaluation; 

adding said new message alternatives to the set of available messages to be 
optimized; and 

choosing the best message alternative from among the choices presented and 
proactively creating message alternatives that are expected to perform well withm some 
confidence level. 



17. A method as in claim 16, wherein said multiattribute system is of standard form. 
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18. A method as in claim 16, wherein said muhiattribute system is reduced to standard 
form. 

5 19. A method as in claim 16, wherein said processing includes discounting said 
performance data; and said discounting being achieved using at least one of 
discounting scheme selected from the set of discounting schemes consisting of: a 
geometric discounting scheme, a linear weighting discounting scheme, a non-linear 
weighting discounting scheme, and combinations thereof. 

10 

20. A method as in claim 1, wherein said processing includes discounting said 
performance data; and said discoxmting being achieved using at least one of 
discounting scheme selected from the set consisting of: a geometric discounting 
scheme, a Hnear weighting discounting scheme, a non-linear weighting discounting 

1 5 scheme, and combinations thereof 

21. A method as in claim 19, wherein said processing includes estimating the distribution 
of the expected payoff of future performance of said message alternatives. 

20 22. A method as in claim 21, wherein said estimate is obtained under the assumption that 
the underlying performance model implied by said multiattribute system holds. 

23. A method as in claim 21, wherein said estimate is obtained by first estimating the 
distribution of expected payoff of future performance of said message alternatives 
25 assuming said underlying performance model unplied by said multiattribute system 
holds, then updating the second said estimate to remove said assumption. 
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24. A method as in claiin 21, wherein said estimate is employed for the purpose of 
allocating future campaign impressions amongst said message alternatives to 
optimize future expected campaign performance. 

5 25. A method as in claim 19, wherein said allocation method comprises performing a 
pairwise comparison procedure among said plurality of message alternatives. 

26. A method as in claim 25, wherein said pairwise comparison comprises: 

Q computing an expected value of a payoff distribution of each message alternative 

i::? 10 and an error in said payoff distribution expected value; and 

\U 

'1* comparing in pairwise manner the expected value based statistic of a selected 

message alternative with the same expected value based statistic for each other of the 
^ mess^e alternatives. 

ni 

s| 15 27. A method as in claim 26, wherein said pairwise comparison comprises: 

•CI 

h'^ selecting a first message alternative and a second message alternative from among 

said plurality of message alternatives; 

computing a quotient determined as difference between the estimate of the said 
expected value divided by a standard error in the difference between the estimate of the 
20 expected value for each of said first and second message alternatives; and 

determining if said difference is within a predetermined magnitude relationship 
of, including greater than, a threshold cutoff value Ci, and: (i) if said difference is within 
said predetermined magnitude relationship of said threshold cutoff value, then identifying 
the second selected message alternative as a member of a non-contending message class. 

25 

28. A method as in claim 27, further comprising: 
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assuming that each message alternative is a content; 

repeating said selecting, said computing a quotient, said computing a difference, 
and said determining for other all other pairs of said message alternatives on a pairwise 
basis so that when all possible pairwise comparisons have been made each message 
alternative will have been sorted mto one contender and said non-contender classes. 

29. A method as in claim 28, fiirther comprising performing said repeating for all other 
pairs of message alternatives on a pairwise basis for each optimization. 

30. A method as in claim 28, wherein said pairwise comparison procedure further 
comprises mitially uniformly allocating said message alternatives prior to performing 
a first optimization. 

31. A method as in claim 27, wherem said allocation algorithm comprises a pairwise 
comparison, said message alternatives are sorted into a contender class and a non- 
contender class, and for at least one identified stage in a messaging campaign all 
visitors are allocated equally among message alternatives belonging to said contender 
class, and no visitors are allocated to message alternatives belonging .to said non- 
contender class. 

32. A method as in claim 31, wherein said at least one identified stage comprises an 
initial stage. 

33. A method as in claun 31, wherein said at least one identified stage comprises a 
plurality of stages. 
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34. A method as in claim 31, wherein said at least one identified stage comprises all 
stages. 

35. A method as in claim 25, wherein said allocation algorithm comprises a pairwise 
5 comparison, and message alternatives are sorted into more than two classes. 

36. A method as in claim 35, wherein said messages belonging to each said class are 
assigned different numbers of visitors. 

10 37. A method as in claim 35, wherein said messages belonging to each said class are 
assigned substantially the same numbers of visitors. 

38. A method as in claim 25, wherein said allocation algorithm comprises a pairwise 
comparison, and message altematives are sorted into more than two classes and the 

1 5 allocation among contenders is not equal. 

39. A method as in claim 25, wherein said allocation algorithm comprises a pairwise 
comparison, and message altematives are sorted into more than two classes and the 
allocation among non-contenders is not equal. 

20 

40. A method as in claim 25, wherein said allocation algorithm comprises a pairwise 
comparison, and message altematives are sorted into more than two classes and the 
allocation among contenders is not equal and the allocation among non-contenders is 
not equal. 



25 
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41. A method as in claim 27, wherein said allocation algorithm comprises a Pairwise 
Comparison, and the cutoff ci is in the range substantially between 0-1 .0. 

42. A method as in claim 27, wherein said allocation algorithm comprises a Pairwise 
Comparison, and the cutoff ci is substantially 0.5. 

43. A method as in claim 27, wherein said allocation algorithm comprises a Pairwise 
Comparison, and the cutoff ci is greater than or equal to L 

44. A method as in claim 27, wherein said allocation algorithm comprises a Pairwise 
Comparison, and the cutoff ci is greater than 1. 

45. A method as in claim 16, wherein reports on the optimal value for each attribute are 
provided only if it is determined that said multiattribute system supports said 
measurement of the relative importance to said message alternative performance of 
said attributes and said attribute values. 

46. A method as in claim 45, wherein said multiattribute system is reduced to standard 
form to determine if said multiattribute system supports said measurement of the 
relative importance to said message alternative performance of said attributes and 
said attribute values. 

47. A method as in claim 45, wherein determination of said relative unportance to said 
message alternative performance of said attributes and said attribute values includes 
determination of confidence intervals around said relative importance. 



A-71214/RMA 



-62- 



48. A method as in claim 16, wherein reports on the optimal value for each attribute are 
provided only if it is determined that said multiattribute system and said performance 

data warrant said measurement of the relative importance to said message alternative 
performance of said attributes and said attribute values. 

49. A method as in claim 48, wherein an acceptance/rejection test is employed to 
determine if said multiattribute system and said performance data warrant said 
measurement of the relative importance to said message alternative performance of 
said attributes and said attribute values. 

50. A method as in claim 48, wherein reports on said relative importance to said message 
alternative performance of said attributes and said attribute values are provided to the 
marketing manager or other interested party, 

51. A method as in claim 48, wherein determination of said relative importance to said 
message alternative performance of said attributes and said attribute values includes 
determination of confidence intervals around said relative importance. 

52. A method as in claim 50, wherein said performance measurement represents the 
click-through rates of the message alternatives, said multiattribute function is the 
logodds function, and said reported relative importance to said message alternative 
performance of one of said attribute values is computed as I00*(exp{w}-1), where 
said u represents a parameter estimate of a multiattribute parameter associated with 
said attribute value, and said reported relative importance to said message altemative 
performmce of one of said attributes is computed as 100x(exp{M}-l), where said u 
represents the largest parameter estimate of a multiattribute parameter associated with 
said attribute values of said attribute. 
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53. A method as in claim 52, wherein said reports include a chart and a table. 

54. A method as in claim 53, wherein said chart is a bar chart where the length of the bars 
are determined by said reported relative importance of said attributes or said attribute 
values. 

55. A method as in claim 54, wherein the colors of said bars are determined by assigning 
each attribute a color, and whenever an attribute or a value of said attribute appears 
on a chart, coloring said bar with said color. 

56. A method as in claim 54, wherein the colors of said bars are determined by, when 
reporting on an attribute, using one color if the lower confidence bound of the best 
attribute value for said attribute is less than zero, and using a second color otherwise; 
and likewise when reporting on an attribute value, using one color if the lower 
confidence bound of said attribute value is less than zero, and using a second color 
otherwise. 

57. A method as in claim 54, wherein the colors of said bars are determined by assigning 
each attribute a color and each attribute value a variant of the color assigned to its 
attribute, and whenever an attribute or an attribute value appears on a chart, coloring 
said bar with said color. 

58. A computer program product for use in conjunction with a computer system, the 
computer program product comprising a computer readable storage medium and a 
computer program mechanism embedded therein, the computer program mechanism 
comprising: a program module that directs the computer system and/or components 
thereof, to function in a specified manner to provide automated measurements of the 
importance of attributes and attribute values of message alternatives and to, in an 
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automated manner, preferentially in real-time, improve the stage-to-stage 
performance of said message alternatives in a multi-stage message campaign in an 
interactive measurable mediixm, the program module including instruction for; 

obtaining historical information including performance data for message 
performance for at least one previous stage of said campaign; 

generating a multiattribute system that describes said message alternatives in 
terms of said attributes and said attribute values; 

determining the relative importance of said attributes and said attribute values 
to the performance of said message alternatives; 

optimizing campaign performance through allocation of message alternatives 
to visitors during said message campaign; and 

determining a next allocation for each new visitor for a next stage of the 
campaign. 

59. A computer program product as in claim 58, wherein said program module further 
including instructions for: 

processing said historical data to identify and correct any erroneous data; 

discounting at least a portion of said performance data from said at least one 
previous stage to grant more weight to more recent performance data than to less recent 
performance data; and wherein: 

said historical information further includes additional information selected from 
the set consisting of: a visitor identifier, a message alternative identifier for the message 
alternative shown to said visitor, a type of action or actions taken by said visitor, and a 
payoff for each of the actions taken by said visitors, a message type, a message type and a 
corresponding altemative clicked on by a visitor, a web site and a section where the 
visitor was presented and saw a banner ad, a time of the visitors visit, a visitor 
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demographic information, a visitor psychographic information, a visitor demographic 
profile, a visitor psychographic profile, and combinations thereof. 

60. A computer program product as in claim 58, wherein said message alternatives are 
selected from a group consisting of: an email, an ad, a banner ad, a splash page, a 
home page, a jump page, a landing page, media programming, media content, a 
political campaign message, a survey, a poll, a news headline, a headline, a ballot 
measure, a ballot initiative, one or more sports scores, one or more sports scores for a 
local, regional, collegiate, or amatexjr sports team or teams, and combinations thereof. 

61. A system for providing automated measurements of the importance of attributes and 
attribute values of message alternatives and for, in an automated manner, 
preferentially in real-time, improving the stage-to-stage performance of said message 
alternatives in a multi-stage message campaign in an interactive measurable medium, 
said system comprising: 

historical information gathering means for obtaining historical information 
including performance data for message performance for at least one previous stage of 
said campaign; 

multiattribute system generation means for generating a multiattribute system that 
describes said message altematives in terms of said attributes and said attribute values; 

first processing means for determining the relative importance of said attributes 
and said attribute values to the performance of said message altematives; 

second processing means for optimizing campaign performance through 
allocation of message altematives to visitors during said message campaign; and 

message altemative allocation means for determining a next allocation for each 
new visitor for a next stage of the campaign. 
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62. A system as in claim 61, wherein said first and second processing means further 
comprise a server having a processor and associated memory. 



63. A system as in claim 62, wherein said system further comprises input/output means 
including a connection to a network. 



1037451 



